This invention pertains to product testing, and particularly to methods and systems for managing product testing programs and product testing program information.
Many products require extensive testing during product development. For example, computer network hardware and software must be tested under real or simulated network conditions for compatibility with legacy hardware and software, for proper operation under a variety of network load conditions, and for proper behavior in response to a variety of network fault conditions. In order to execute these and other tests for a product, an engineer typically develops a test plan, designs a variety of tests to meet the objectives of the plan, and executes these tests.
When a new product is to be tested, a test engineer manually produces a test plan document, e.g., by writing the document using a computer-based text/document editor. For each test in the plan, the engineer executes the test one or more times and records the results. Recording generally consists of entering results in a lab notebook, making notations in a printed copy of the test plan, copying screen output to files, etc. Once the testing is complete, the engineer manually produces a test report that summarizes the test results in the context of the test plan. The production of this report is typically time-consuming, and is considered to be drudgery by many test engineers.
In some circumstances, tests can be executed automatically. With automatic testing, a test engineer must still develop a test plan and a test report. In addition, the engineer develops automated tests. The benefit of such an approach is that tests, particularly those that require multiple instances, can then be executed without manual intervention, with test results stored automatically to result logs, screen dump files, etc.
Whether tests are executed manually or automatically, several difficulties remain. Although a test summary report may refer to the location of the detailed results, these results can be stored in as many different formats and locations as there are test engineers executing tests. Further, these results are typically not visible to anyone other than the test engineer prior to the completion of the report. These problems make it difficult for anyone to track the testing status of large programs that may span several groups or projects.
The present invention provides a solution to the test-tracking and result visibility problems, and produces a variety of other benefits as well. Generally, the invention comprises a web-based interface to a database for housing and cross-linking test plans, test suites, tests, and test results. A test engineer develops test plans, test suites, and tests using a tester""s web interface, and updates the database with test results when tests are executed (this step can be automated). A second web interface, the query interface, allows others to have visibility into the database contents. This interface allows users to query the system based on several attributes and then view or print test plans, suites, tests, and results matching these attributes.
A test information management system according to the invention can be much more than a repository for test resultsxe2x80x94it allows the integration of test planning, test execution, and results tracking. The tester""s web interface can provide guidance to inexperienced testers, showing them the types of things that they should be thinking about when planning and executing tests. It also allows testers to view, emulate, or copy other tester""s  plans and tests from other projects that may be applicable to a new project, thus encouraging re-use and consistency. Developers can also use the system to reproduce a test failure without lengthy help from the engineer who conducted the test. For managers and other results trackers, the query interface allows them to construct an up-to-date xe2x80x9cbig picturexe2x80x9d of a project, as well as delve into the specifics of a single test by navigating easily comprehendible web hyperlinks. And for employees working with customers, the query interface allows them to identify similar product configurations, how they were tested, what problems were found and how they were fixed, etc.
Visibility into development testing can even be extended to the customers themselves. This aspect of the invention integrates two normally separate organizational functions-product development testing and product support. Customers can browse tests related to their products, observing how the products were configured for testing, what features were tested and the results of those tests, how bugs were resolved, etc. This information can be valuable to customers, and with the present invention, requires almost no extra effort to provide.
In a first aspect of the invention, a centralized test information management system is disclosed. The system comprises a database for storing test plan data, test data, and test result data for multiple products. The system further comprises two web interfaces-a tester""s web interface and a query web interface. The tester""s web interface presents a navigable collection of templates that allow testers to develop test plans and tests, enter test results, and associate test plans, tests, and test results. Tester input is taken from these templates and stored in the database. The query web interface allows remote users to enter queries and retrieve data matching those queries from the database. The query web interface presents the retrieved data as viewable, and associated, test plans, tests, and test results.
In another aspect of the invention, a method of tracking an ongoing test program is disclosed. This method comprises using a web-based application to gather data elements for a test plan, a test, and test results, and associate the results, the test, and the test plan, the application storing data elements representing the test plan, test, results, and association in a database. The method further comprises using a second web-based application to retrieve data elements from the database in response to a user query and generate a web page for a desired test plan or test using the data elements.
In a third aspect of the invention, a method of supplying product information to customers and/or customer account representatives is disclosed. When a product is tested, the tests and test results are compiled into a searchable database (preferably by the testers themselves during the testing process) keyed to one or more product attributes. A web-based search utility allows users to search the database for test data matching a selected attribute value. The search utility returns the results in a viewable web page.